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ABSTRACT 

> A technique  for  reducing  corporate  judgement  to  a mathematical 
model  is  presented.  This  work  is  an  extension  of  Judgement 
Analysis/Policy  Capturing. 
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INTRODUCTION 


Techniques  are  in  the  literature  to  model  on-going,  corporate 
decision  making  processes  (Policy  Capturing)  and  the  decision 
making  process  which  may  be  applied  to  a system  (Judgement 
Analysis).  As  Ward  has  documented  (#1),  the  doctoral  admission 
policies  of  the  University  of  North  Dakota  have  been  success- 
fully captured  and  quantified.  In  still  another  context,  t.he 
decision  process  of  the  US  Supreme  Court  has  been  reduced  to 
an  equation  based  on  previous  decisions  in  similar  cases. 

In  application.  Judgement  Analysis  requires  the  ranking  of 
conflicting  configuration  of  the  entity  or  process  of  which  a 
mathematical  model  is  needed.  For  example,  most  respondents 
would  highly  desire  a job  which  paid  $40 , 000/annum  for  a stand- 
ard 40  hour  week.  Given  the  choice,  few  would  opt  for  $30,000/ 
annum  for  the  same  week.  However  some  may  prefer  $30k  for  a 
30  hour  week,  or  a 20  hour  week  and  certai.nly  most  would  opt 
for  a 2 hour  week  at  $30K/annum.  Likewise,  others  may  take 
$45K  for  a 50  hour  week  and  surely  most  would  probably  jump  at 
S150k  for  a 60  hour  week.  Judgement  Analysis  provides  the 
method  to  find  the  coefficients  for  the  equation; 

Job  Desirability  = Aq+  A^*salary  + .A  2 * workweek 

+A3*salary^+.A4  workweek^ 

+A5*salary *workweek+ 

There  are,  however,  problems  of  magnitude  in  application. 

Consideration  of  biographic  data  in  attempting  to  build  an 
enlisted  procurement  model  led  to  tens  of  thousands  of  com- 
binations of  attributes.  Prospective  enlistees  have  four  test 
scores  each  of  which  may  take  on  19  values,  one  test  score  in 
the  internal  of  0-100,  courses  taken  in  high  school,  age, 
number  of  dependents,  and  numerous  other  personal  attributes. 
All  possible  combinations  rapidly  approaches  a googol.  Judge- 
ment Analysis  was  attempted  using  stochastic  sampling  and  was 
not  successful.  This  had  led  to  an  extrapolation  of  Judgeme.nt 
Analysis  ...  Judgement  Specification. 


METHOD 


To  overcome  the  problem  of  magnitude  generated  by  the  enlisted 
procurement  problem,  an  analysis  of  the  process  used  by  the 
"expert"  respondents  was  initiated.  (Recall  that  random 
samples  were  used) . Typically  the  "expert"  would  select  his 
"best"  enlistee  and  his  "worst"  enlistee.  Then  his  second 
best  and  second  worst.  And  so  forth  until  he  exhausted  his 
300  candidates.  This  process  in  the  absence  of  complete 
information  (the  other  2,700  configurations)  resulted  in  five 
diverse  models  from  five  "experts".  Independent  interviews 
and  then  joint  interviews  with  the  respondents  revealed  some 
common  trends  which  were  probably  imbedded  in  the  diverse  models. 
Comments  such  as  "for  a computer  programming  job,  it  doesn't 
matter  what  the  man's  mechnical  aptitude  is,"  or  "for  computer 
programmers,  I don't  care  how  old  the  man  is  as  long  as  he 
is  over  17  and  under  22."  And,  "the  last  thing  I want  is  a man 
who  is  not  a high  school  graduate".  From  this  it  was  concluded 
that  working  independently  with  sample  data  our  experts  agreed 
to  disagree.  But  when  used  as  a corporate  consultant,  it  was 
found  that  the  e.xperts  could  accommodate  one  another  to  arrive 
at  a mutually  agreed  upon  threshold  where  they  would  give  up  a 
desirable  attribute. 

Judgement  Specification  then  requires  the  brokeraging  of  a 
corporate  body  of  experts  to  arrive  first  at  the  most  desirable 
configuration  of  the  entity  to  be  modeled  and  then  the  order  in 
which  attributes  will  be  given  up.  This  manual  model  is  then 
converted  into  a mathematical  model  or  an  equation  which  can  be 
used  to  rank  any  possible  configuration  of  the  entity  under 
consideration. 

The  enlistment  question  of  com.puter  programmers  was  posed  to  a 
body  of  five  senior  enlisted  programmer  personnel  under  the 
supervision  of  a moderator  - "given  the  following  attributes 
will  be  used,  how  would  you  define  the  best  possible  enlistee 
to  be  assigned  as  a programmer  trainee?" 


AGE 

AFQT 

ED  LEVEL 

IDEPENDENTS 

17 

30 

< High  School 

0 

13 

31-40 

High  School 

1 

19 

41-50 
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» 

Some  College 

2 

20 
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21 

22 
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>3 
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45* 

0 

0 

0 
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50 

5 

5 

5 

55 

1 

10 

1 

10 

10 

i 

1 

1 

95 

95 

95 

1 

95 

*minimum 

score  of 

45  required 

on  General  Airman 

Qualif 

ying 

Exam  for  entry  into  the  Air  Force. 

Then  by  a process  of  negotiated  attrition,  "how  would  the  corporate 
body  yield  these  attributes,  i.e.,  of  all  of  the  attributes 
which  will  be  given  away  first  and  how  much,  until  all  attribute 
values  have  been  yielded?" 

In  this  case,  t.he  ideal  programmer  trainee  was  defined  by  the 
attribute  values  listed  first  under  each  attribute  shown  in 
the  table  below.  The  Judgement  Specification  panel  then  yielded 
the  attribute  values  secuentiallv  in  the  order  shown  below: 


AGE 

I ^ 

' S20 

^21 

. >22 
“^^<>22 
‘*‘♦<17 


AFQT 
91-100 
81-90 
^ 71-80 
? 61-70 


^^>71-8  0 


ED  LEVEL 
. Some  College 
p Degree 
pHigh  School 
^<High  School 


^DEPENDENTS 

1 <•  T 
2 

H5-<J 


CURRICULUM 


^College  Prepatc 

^General 

J / ^Vocational 


APTITUDE  SCORES 


GEN 

>>90 

'^<85 

H<8  0 


ADM 

';S90 

''>85 
"'<80 
•^>7  5 
^'^7  0 
6 5 


MECH 

> 90 
. >85 
! >80 

5>70 
,>6  5 


ELECT 
li  <-  95 

"<35 

“;>s 

^-6  0 


*Mandatory  minimum  to  enter 
proarammina  career  field. 


These  numbers  were  then  transformed  to  reflect  the  relative 
weight  attached  to  the  individual  attributes  and  the  relationship 
within  each  attribute.  This  was  done  by  assigning  the  hiahest 
ordinal  value  at  which  yielded,  to  the  most  desirable  value 
in  the  attribute  column  and  employing  the  delta  at  each  step 
thereafter  to  generate  the  re.maining  values.  The  following 
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payoff  coefficients  resulted: 


AGE 

AFQT 

ED  LEVEL 

# DEPENDENTS 

19:44 

91-100:46 

Some  College:  51 

1:45 

20:43 

81-90: 37 

Degree:  36 

0:43 

21:28 

71-80:17 

High  School:  31 

2:24 

18:27 

61-70.11 

^High  School:  0 

3:9 

22:26 

51-60:8 

>3:0 

22:  1 

< 51:0 

17:0 


CURRICULUM 

APT  I 

TUDE  SCORES 

College  Prepatory:  37 

GEN 

ADM 

MECH 

ELECT 

General:  7 

95:49 

95:47 

95:42 

95:50 

Vocational:  0 

90:37 

90:37 

90:39 

90:38 

85:30 

85:36 

85:38 

85:29 

80:26 

80:33 

80:37 

80:27 

75:22 

7 5:22 

75:36 

75:23 

70:16 

70:21 

70:35 

■'0:17 

65:9 

65:15 

65:34 

65:9 

60:0 

60:8 

60:11 

60:2 

<60:0 

<60:0 

<60  : 0 

The  above  produces  a mutually  exclusive  categorical  model 
which  returns  payoffs  in  the  interval  of  O-to-411.  This 
score  may  be  converted  to  any  convenient  interval,  say  O-to-100, 


airmen  where  "success"  was  defined  as  achieving  paygrade  E-5 
(Staff  Sergeant)  in  less  than  the  median  time  of  55.5  months.  t 

These  methods  were:  1.  Multivariate  Regression,  2.  Odds  for  | 

Effectiveness  and  3.  Judgement  Specification.  The  results  of 
this  investigation  were  documented  in  DPMDDA  Memorandum, 

"PRO.MIS  Person-Job-.Match  Algorithm  Predicting  Job  Success",  dated 
30  March  1976,  which  is  included  as  Appendix  A.  This  memorandum 
concluded,  _ _ the  predictive  equations  derived  from  the 

three  methods  provrde  similar  degrees  of  accuracy  with  regression 
and  Odds  for  Effectiveness  being  slightly  better  than  Judgement 
Specification.  One  would  thus  conclude  that  the  decision  on 
which  technique  to  employ  would  have  to  be  made  on  considerations 
other  than  oredictive  accuracv  (i.e.  cost)." 


CONCLUSION 


Judgement  Specification  is  the  indicated  technique  when  n-jmerous 
attributes/values  must  be  included.  For  example,  the  programmer 
trainee  discussed  above  could  be  defined  in  14,696,640  different 
configurations.  Not  feasible  in  Judgement  Analysis.  The  process 
is  much  faster,  simpler  and  cheaper  than  the  attendent  "card 
sorting"  involved  in  Judgement  Analysis.  Further,  all  expert 
respondents  will  understand  the  process  and  will  be  forced  to 
consider  the  relative  worth  of  every  attribute  value  resulting 
in  one  brokered  equation  for  the  panel  of  experts.  Finally, 
the  Judgement  Specif iaction  Technique  will  provide  a non-linear 
m.odel  without  the  additional  cost  of  model  seeking. 

Drawbacks,  however,  do  e.xist.  Without  a doubt  the  model  will 
be  extremely  sensitive  to  which  experts  comprise  the  panel.  Also 
it  is  probable  that  a persuasive  or  weak  moderator  could  bias 
the  model,  or  possibly  a "natural-born"  leader  on  the  panel 
could  subvert  the  model  to  reflect  "his"  model  of  the  process 
under  study. 


MEMORANDUM 


PROMTS  PERSON-JOB-:4ATCH  ALGORITHM 
PREDICTING  JOB  SUCCESS 

SUMMARY  & CONCLUSIONS  OF  DEVELOPMENT  TO  DATE 

1.  ODDS  FOR  EFFECTIVENESS  - This  technique  is  especially  use- 
ful in  developing  prediction  equations  when  the  dependent 
variable  to  be  predicted  is  dichotomous  in  nature  (e.g.  pass/ 
fail,  attr it/continue , etc.).  T.he  procedure  consists  of  (a) 
collecting  an  historical  data  base  containing  predictive  at- 
tributes and  the  value  of  the  dichotomous  dependent  variable 
(usually  coded  0 or  1)  for  each  individual  in  the  data  base; 

(b)  defining  a series  of  categories  of  individuals  based  on 
predictive  attribute  values  (c)  computing,  for  each  category, 
the  proportion  of  i.ndividuals  in  that  category  who  have  a 
particular  value  of  the  dependent  variable  (the  proportion  thus 
computed  for  eac.h  category  then  becomes  the  predicted  probability, 
or  odds  that  an  i.ndividual  whose  attribuces  place  him  in  t.hat 
category  will  take  on  the  particular  value  of  the  dichotomous 
dependent  variable) . (d)  multiple  linear  regression  is  then  used 

to  convert  the  Odds  for  Effectiveness  table  to  a single  equation 
in  which  an  individual's  attribute  values  are  used  to  compute 

his  predicted  odds. 

2.  APPLICATION  TO  PROMIS  - In  previous  studies  we  had  selected 
time  for  promotic.n  to  Staff  Sergeant  (E-5)  as  a measure  of  job 
success  and  attempted  to  run  regression  to  predict  when  an 
individual  would  make  E-3.  Like  the  HRL  regression  studies 

Appendix  A 
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(to  predict  and  individual's  grade  in  Tech  School)  we  obtained 
very  low  R-squared  values (on  the  order  of  .15).  Using  the 
same  data  base  (E-5's  currently  serving  in  the  702XX  specialty 
who  entered  the  Air  Force  since  19G6,  were  net  cross  trainees 
or  prior  service)  we  applied  the  Odds  for  Effectiveness  technique 
b'  creating  a dichotomous  dependent  variable  in  which  an  "ef- 
fective" airman  was  defined  as  one  who  was  promoted  to  E-5  in 
less  than  the  median  time  of  55.5  months  (coded  1).  The 
categories  were  then  constructed  and  each  airman  in  the  data 
base  was  placed  in  a category  based  on  .his  attributes.  The 
probability  of  making  E-5  in  less  than  55.5  months  (i.e.  being 
"effective"),  was  then  computed  for  each  category.  This  infor- 
mation was  then  put  into  equation  form  by  running  regression; 
the  resulting  equation  had  a very  i.mpressive  R-squared  value  of 
approximately  .3. 

3.  TESTING  - At  this  point  in  our  analysis  we  had  three  tech- 
niques, eac.h  of  which  oroported  to  predict  job  success  in  the 
702XX  specialty: 

a.  REGRESSION  - Essentially  the  same  technique  employed 
by  HRL;  very  low  R-squared. 

b.  JUDGEMENT  SPECIFICATION  - The  quantified  judgement  of 
five  senior  people  with  much  experience  in  the  702XX  specialty. 

c.  ODDS  FOR  EFFECTIVENESS  - Same  data  used  as  the  regression 
technique  but  high  R-squared. 

To  evaluate  these  equations  the  following  scheme  was  employed: 

a.  Of  the  overall  data  records  available,  80%  were  selected 
at  random  (about  2600  records)  for  use  in  constructing  the  re- 


qressioti  atui  Odds  t'Oi'  K r’ r |■‘c t. i vonoss  {‘(jiki  t- i oiis  (t  lio  Judijcment 
Specification  eqviation  is  not  iiased  on  historic-al  data). 

b.  T!ie  remaining  20 A of  the  data  (about  7 00  records)  was 
used  to  tost  the  predictive  accuracy  of  the  equations. 

c.  Each  record  in  the  te.st  data  base  was  scored  by  each  of 
the  three  equations. 

d.  Each  recoi'd  in  the  tt'st  data  l'as<'  wms  tlien  I'valuatt'd  as 
being  a "success"  or  "failure"  based  on  when  the  individual  was 
promote!,!  to  E-5  (below  or  above  the  median). 

e.  series  of  cut  score.s  (or  critical  values)  were  then 
selected  for  oacl;  of  the  equations.  The  cut  score  providi'd  an 
arbitrary  definition  of  predicted  "succes.s"  (i.e.  if  the  consputed 
score  is  equ.il  to  or  greater  than  the  cut  score,  the  record  is 
predicted  to  be  a "success"). 

f.  For  each  cut  score  for  each  equation  t.v/o  percentages 
were  computed:  The  percent  of  the  test  data  predicted  to  be 
successes;  and  the  percent  of  tiio  prediett'd  successes  who  are 
actually  successes  (measure  of  accuracy) . 

g.  These  two  percentages  wore  theit  plotted  for  each  cut 
score  for  each  of  the  three  equatior\s  being  evaluated  (See 
Figure  1)  . 

h.  Figure  2 is  a graphical  representation  of  the  relation- 
ship between  critical  value,  percent  predicted  succes.s,  and 
percent  accuracy  for  tite  Odds  for  Ef  feet  ivencss  equation  as 
applied  to  the  test  data  base. 


4.  CONCLUSIONS  - Figure  1 shows  that  the  predictive  equations 
derived  from  all  three  techniques  provide  very  similar  degrees 
of  accuracy  with  regression  and  Odds  for  Effectiveness  being 
slightly  better  than  Judgement  Specification.  One  would  thus 
conclude  that  the  decision  on  whcih  technique  to  employ  would 
have  to  be  made  on  considerations  other  than  predictive  accuracy 
(i  .e . cost) . 

5.  RECOMMENDATION  - Since  HRL  has  already  begun  development  of 
job  success  predicting  equations  using  the  regression  technique, 
it  would  not  be  cost  effective  to  pursue  Judgement  Specification 
or  Odds  for  Effectiveness  any  further  for  application  to  PROMIS. 


